Industrial virtual assistant platform with robotic process automation for knowledge and insights management

ABSTRACT

An Industrial Virtual Assistant (IVA) platform with Robotic Process Automation that operates like a Digital Knowledge Companion and allows operational staff at industrial facilities to have natural language conversations with the IVA to obtain information about, and to control operations of, industrial facilities, and which automates certain processes based in part on those natural language conversations. In an embodiment, the platform uses a Robotic Process Automater (RPA) to ingest information from documentation, human inputs, and operational data from the facility, organize that information into a knowledge graph containing comprehensive facility information, and apply machine learning algorithms to the knowledge graph to provide natural language responses to human queries and to automate certain processes of the facility.

CROSS-REFERENCE TO RELATED APPLICATIONS

Priority is claimed in the application data sheet to the followingpatents or patent applications, each of which is expressly incorporatedherein by reference in its entirety:

Ser. No. 17/953,924

Ser. No. 17/700,407

63/307,595

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of process automation, and moreparticularly to the field of process automation in industrialfacilities.

Discussion of the State of the Art

The utility and industrial sectors are experiencing a decline insufficiently knowledgeable operational staff which is limitingday-to-day operations in these industries, leading to loss of revenueand potentially damaging the public image of the entities involved.While there are a number of reasons for this decline, a major cause isthe retirement of older staff whose breadth of operational knowledge isoften lost upon their retirement. Even where documentation exists, agreat deal of real-world operational knowledge of the particularfacility (e.g., finicky pumps, undocumented heat issues, etc.) is oftenlost.

Further exacerbating the operational knowledge problem is the fact thatall facilities more than a few years old rely on older and/or outdatedtechnology. Older software tools for process controls and otherfacilities operations are not intuitive in their setup and operation.Their user interfaces are limited to traditional tools such as graphs,dashboards, and complex menu systems. Even newer facilities with modernsoftware have complicated user interfaces that require detailedknowledge to operate. Moreover, these interfaces are inflexible,requiring extensive training and background knowledge to navigatethrough them to find the relevant controls and information.

Additionally, systems in industrial facilities that detect errors andanomalies have simplistic, hard-coded rules that must be properlyinterpreted by domain experts to determine the cause and solution.Depending on the type and complexity of the incident, many differentsets of rules may be applicable, and the domain expert must know whichrules apply. This type of operational knowledge is exceedingly difficultto transfer to less experienced operational staff, andcurrently-available knowledge systems are incapable of adapting to thelevel of complexity required to assist in the operation of industrialfacilities.

Such users would rather be served better if they can have an intuitive“conversation” with the system to “inform” of new situations that shouldbe taken into consideration when detecting incident anomalies andproviding insights, as well as automating certain processes based on acombination of documentation, natural language user interactions, andreal-time operational data from the system.

To overcome these operational knowledge difficulties, what is needed isan Industrial Virtual Assistant (IVA) platform with robotic processautomation that allows operational staff at industrial facilities tohave natural language conversations with the IVA to obtain informationabout, and to control operations of, industrial facilities, and whichautomates certain processes based in part on those natural languageconversations.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, anIndustrial Virtual Assistant (IVA) platform with Robotic ProcessAutomation that operates like a Digital Knowledge Companion and allowsoperational staff at industrial facilities to have natural languageconversations with the IVA to obtain information about, and to controloperations of, industrial facilities, and which automates certainprocesses based in part on those natural language conversations. In anembodiment, the platform uses a robotic process automater (RPA) toingest information from documentation, human inputs, and operationaldata from the facility, organize that information into a knowledge graphcontaining comprehensive facility information, and apply machinelearning algorithms to the knowledge graph to provide natural languageresponses to human queries and to automate certain processes of thefacility.

According to a preferred embodiment, a robotic process automation systemis disclosed, comprising: a computing device comprising a memory, aprocessor, and a non-volatile data storage device; a knowledge graphstored on the non-volatile data storage device, wherein the knowledgegraph comprises nodes representing things and edges representingrelationships between the things; a robotic process automater comprisinga first plurality of programming instructions stored in the memorywhich, when operating on the processor, causes the computing device to:receive data about a facility to be automated, the data comprising: oneor more human inputs; one or more documents; and one or more operationaldata streams from the system; store the received data in the knowledgegraph; process a portion of the knowledge graph through a domain machinelearning algorithm to obtain an inference about the facility notcontained in the received data; store the inference as part of theknowledge graph; receive a query about the facility related to theinference; process the query and a portion of the knowledge graphthrough a context recognition machine learning algorithm to determine acontext in which the query was made; process the context and a portionof the knowledge graph through the domain machine learning algorithm to:generate a response to the query; and determine whether the inferencecan be generalized to the context of the query; and store thedetermination as part of the knowledge graph.

According to another preferred embodiment, a method for operating arobotic process automation system is disclosed, comprising the steps of:storing a knowledge graph a non-volatile data storage device of acomputing device comprising a memory, a processor, and a non-volatiledata storage device, wherein the knowledge graph comprises nodesrepresenting things and edges representing relationships between thethings; using a robotic process automater operating on the computingdevice to: receiving data about a facility to be automated, the datacomprising: one or more human inputs; one or more documents; and one ormore operational data streams from the system; store the received datain the knowledge graph; process a portion of the knowledge graph througha domain machine learning algorithm to obtain an inference about thefacility not contained in the received data; store the inference as partof the knowledge graph; receive a query about the facility related tothe inference; process the query and a portion of the knowledge graphthrough a context recognition machine learning algorithm to determine acontext in which the query was made; process the context and a portionof the knowledge graph through the domain machine learning algorithm to:generate a response to the query; and determine whether the inferencecan be generalized to the context of the query; and store thedetermination as part of the knowledge graph.

According to an aspect of an embodiment, the query is a natural languagequery from a person and the response is provided as a natural languageresponse using a natural language understanding (NLU) engine.

According to an aspect of an embodiment, the natural languageunderstanding (NLU) engine comprises a natural language processingmachine learning algorithm trained in part on a domain-specific taxonomyfor a domain related to the facility.

According to an aspect of an embodiment, the natural language processingmachine learning algorithm is further trained on information from theknowledge graph to enhance its domain-specific taxonomy recognition.According to an aspect of an embodiment, the query is an automated queryfrom a component of the facility and the response is a controlrecommendation sent to a supervisory control and data acquisition(SCADA) module to generate a control signal.

According to an aspect of an embodiment, the information from theinferences about the facility stored in the knowledge graph are analyzedover time to identify long term patterns which have not been provided bythe received data about the facility.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several aspects and, together withthe description, serve to explain the principles of the inventionaccording to the aspects. It will be appreciated by one skilled in theart that the particular arrangements illustrated in the drawings aremerely exemplary and are not to be considered as limiting of the scopeof the invention or the claims herein in any way.

FIG. 1 is a block diagram illustrating integration of a distributedindustrial virtual assistant platform into an industrial facility.

FIG. 2 is a block diagram illustrating an exemplary system architecturefor a distributed industrial virtual assistant platform.

FIG. 3 is a block diagram illustrating an exemplary architecture for thecommunication manager aspect of the exemplary a distributed industrialvirtual assistant platform.

FIG. 4 is a block diagram illustrating an exemplary architecture for theIVA manager aspect of the exemplary a distributed industrial virtualassistant platform.

FIG. 5 is a block diagram illustrating an exemplary architecture for thedomain expertise manager aspect of the exemplary a distributedindustrial virtual assistant platform.

FIG. 6 is a block diagram illustrating an exemplary architecture for theadaptation manager aspect of the exemplary a distributed industrialvirtual assistant platform.

FIG. 7 is an illustration showing a simplified layout of an oil refineryas an example of an industrial facility into which a distributedindustrial virtual assistant platform may be integrated.

FIG. 8 is a method diagram illustrating an exemplary process forre-training of domain and context machine learning algorithms based onuser interactions.

FIGS. 9A & 9B are flow diagrams illustrating an exemplary operationaluse of a distributed industrial virtual assistant platform to assist inthe operation of an industrial facility.

FIG. 10 is a block diagram illustrating an exemplary conceptual mappingarchitecture for a dynamic concept learning system.

FIG. 11 is a flow diagram illustrating a dynamic concept learningprocess.

FIG. 12 is a flow diagram illustrating the workflow from a plurality ofconversations to a generation of an incidents network.

FIG. 13 is a flow diagram illustrating the workflow for using leaningsfrom past user conversations to classify new incidents and provide aricher incident classification experience.

FIG. 14 is a flow diagram illustrating how an industrial virtualassistant learns from unstructured information and adds that to anincident map for future retrieval.

FIG. 15 is a block diagram illustrating an exemplary backendarchitecture process for a distributed industrial virtual assistantplatform.

FIG. 16 is a diagram showing integration of exemplary knowledge sourcesof an industrial facility by a robotic process automater.

FIG. 17 is an exemplary system architecture for a robotic processautomater aspect of an industrial virtual assistant.

FIG. 18 is an exemplary knowledge graph of the type that may be used byan industrial virtual assistant with robotic process automation.

FIG. 19 is a diagram showing an exemplary application of a roboticprocess automater aspect of an industrial virtual assistant.

FIG. 20 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device.

FIG. 21 is a block diagram illustrating an exemplary logicalarchitecture for a client device.

FIG. 22 is a block diagram showing an exemplary architecturalarrangement of clients, servers, and external services.

FIG. 23 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device.

DETAILED DESCRIPTION OF THE DRAWING FIGURES

The inventor has conceived, and reduced to practice, an IndustrialVirtual Assistant (IVA) platform with robotic process automation thatallows operational staff at industrial facilities to have naturallanguage conversations with the IVA to obtain information about, and tocontrol operations of, industrial facilities, and which automatescertain processes based in part on those natural language conversations.In an embodiment, the platform uses a robotic process automater (RPA) toingest information from documentation, human inputs, and operationaldata from the facility, organize that information into a knowledge graphcontaining comprehensive facility information, and apply machinelearning algorithms to the knowledge graph to provide natural languageresponses to human queries and to automate certain processes of thefacility.

The industrial virtual assistant platform assists engineers and otherfacilities staff by simplifying interactions with facility equipment,hardware, and software. Modern industrial facilities have complicatedprocess control systems which require highly trained staff to operate.Many of those process control systems use archaic hardware and software,requiring very specialized knowledge and making interactions with thesystems difficult. A industrial virtual assistant platform simplifiesthe process of interacting with such facilities by allowing naturallanguage querying and control of the facility. For example, utilitieslike water companies and electric companies have many large andexpensive pumps that move water, fuel, coolant, etc. around. Theknowledge in the industrial pumps' domain is highly specialized, butwith a cloud-based IVA pump technicians can ask for the information theyneed when needed without being trained in all the aspects of eachdiffering pump.

Additionally, by learning experientially, the IVA can become therepository of operational wisdom that can be retained through changes offacility staff over time. Rather than asking retiring workers andengineers to manually document their knowledge, the workers andengineers can simply work day-to-day with an IVA that can begin tounderstand how the experienced workers and engineers respond to eachevent. Each parameter of the equipment in the facility can be digitallylogged, every interaction with equipment or change in operationalprocesses can be tracked, and machine learning can used to makepredictions about facility operations. For example, the hardware,software, and situational parameters that lead up to a specific eventcan be processed through machine learning algorithms to understand howto respond to a similar event in the future. In an embodiment, theplatform uses a robotic process automater (RPA) to ingest informationfrom documentation, human inputs, and operational data from thefacility, organize that information into a knowledge graph containingcomprehensive facility information, and apply machine learningalgorithms to the knowledge graph to provide natural language responsesto human queries and to automate certain processes of the facility.

One of the problems that the distributed Industrial Virtual Assistant(IVA) platform is designed to solve is that systems in industrialfacilities that detect errors and anomalies have hard-coded rules thatcannot be easily modified to adapt to changing conditions or processes.In the past, dynamic learning through employing various AI and machinelearning methods has been limited to enhancing detection/predictioncapabilities on individual datasets rather than on enhancing the rulesthemselves that define the criteria for detecting an incident, includingnew unstructured knowledge requiring engineering efforts to enhance theanalytics engine's capabilities. Part of the problem with training suchAI systems for feedback and control of industrial facilities is thatdomain-expert users themselves find it hard to express theirunderstanding of how the incident detection rules should evolve in astructured fashion, as expected by such limited underlying systems.Depending upon how complex the incident and underlying phenomena is,there might be many different sets of rules that are applicable indifferent situations, and domain experts often assimilate and makedecisions intuitively based on their experience in the field and with aparticular facility or its equipment. It is difficult to express suchvariability in rules and yet have the detection system apply the bestrules given specific circumstances.

The distributed Industrial Virtual Assistant (IVA) platform addressesthis problem by having a virtual assistant architecture wherein userscan have natural-language conversations to train the IVA on equipment,domain concepts, and common tasks. This is similar to training a newemployee. The IVA will also learn from a user's behavior and will overtime adapt to personal preferences. The IVA will know what a givenuser's job is, where he or she is located, and what he or she needs on aday-to-day basis. Relevant information will be delivered to the userautomatically. The learning of the user's job will also benefit othercompanies because the data can be anonymized and sent back to the cloudto inform a more general machine learning model that applies to that jobacross the whole sector.

Through adaptive learning of domain specific concepts and contexts(e.g., energy, sewer, electric, and water systems), the IVA can respondto user queries involving such concepts using industry vernacular andnomenclature. For example, a facility manager/engineer might ask (byvoice or text) the industrial virtual assistant that may be installed onhis or her mobile device “What the fluid pressure is at a certainpumping station during time interval x?” In this case, the industrialvirtual assistant has the relevant domain expertise to understand thecorrect meaning of the technical terms: “fluid pressure,” “in thepumping station,” and “at time interval x.” With this understanding, thevirtual assistant can find the relevant asset ID in the facility assetdatabase, locate the relevant fluid pressure sensor ID in the SCADAsystem, and retrieve the relevant fluid pressure data from the sensor(either in real time or from stored data) at the required time point.Thereafter, the industrial virtual assistant will respond to the user(voice or text) with the following answer: “the fluid pressure in pump xin location y and at time z is w psi.” This will also allow theorganization to achieve its operational goals with minimal impactrelated to shortages in skilled labor.

In an embodiment, the IVA platform is initially provided with afacilities asset database containing information about equipment,components, sensors, and processes, and is connected to process controlsoftware and/or equipment control software of the facility such thatinformation about the current status of the facility (e.g., equipmentoperational statuses, sensor data, flow controls, current processes inoperation, etc.) can be accessed by the IVA platform. The facilitiesasset database may contain knowledge not only about the facilitiesassets (e.g., tank capacities, pipe capacities and flow rates, pumppressures, etc.), but also about processes within the facility (e.g.,flow rates are not allowed to exceed a certain rate during certainoperations, increasing processing volumes of a given operation willreduce efficiency by a certain percentage, etc.), such that machinelearning algorithms can be trained to provide context-sensitive anddomain-sensitive responses to user queries based on these databases. Asusers interact with an IVA instance and/or the IVA platform, thesedomain-specific knowledge databases are updated by continual re-trainingof machine learning algorithms based on exceptions and/or incidentsidentified during the interactions. As one example, if a domain-specificknowledge database for pumps indicates that a particular pump should beoperating at a minimum pressure of 100 psi, but an operator determinesthat a given process works better if the pump is operated at 90 psi, thesystem will learn that the preferred operation for that pump is 90 psi,not 100 psi.

The IVA platform may comprise one or more domain-specific knowledgedatabases, each containing rules about that domain of knowledge that canbe modified over time by training and re-training of machine learningalgorithms using voice, text, and visual interactions with users and/orequipment over a period of time to determine which actions and behaviorsgovern incidents. The rules may include checking on one or multipleexceptions, existence of various conditions, simple statistical checkson various data sets and even including static meta information aboutthe assets involved. For each incident type, the IVA may be provideddifferent sets of rules over time. The IVA platform's backenddynamically builds a convergence map that guides it to apply the bestset of rules and make a highly accurate incident detection andclassification for similar future incidents, thereby eliminating thelaborious work by domain experts and operators of having to checkvarious datasets and exceptions manually. The IVA, after selecting thebest set of rules, automatically performs the checks and bringsactionable insights to the users, saving significant amount of timeduring time critical incident response scenarios.

The IVA does more than domain learning, the IVA also learns complexincident behaviors (e.g., in industrial/utility processes). From thesecomplex incident behaviors, the IVA can learn to connect data fromvarious systems and over time to anticipate problems and prescribeactions to users. The IVA can be trained on vendor-specific equipmentallowing companies to expertly craft interactions between an IVAinstance, a given user, and relevant facility assets, equipment, orprocesses.

The IVA may learn during the occurrence of incidents and useexception-based learning (e.g., from users handling an incident notcovered by existing rules). The IVA may have a switching mode wherebyusers can set an IVA instance to engage either active or passivelearning or a combination of the two. The IVA may have a retroactivelearning mode, whereby after an incident, engineers/staff instruct theIVA to go back in time and behave as if live streams of data were comingin, and then provide the IVA with curated responses that the engineersand staff would have taken to minimize the consequences of the event.With enough retroactive training, the IVA can begin to take the samecorrective actions in the future during the same or even similar events.

A cloud-based industrial virtual assistant can be used in any domain andacross many domains simultaneously. Core computing (for the IVAplatform, for example) may be configured to happen in the cloud withdomain-specific and user-specific IVAs for certain users or groups. Formore security-sensitive applications, the IVA platform or IVA instancescan be implemented on a company's intranet or compartmentalized networkssuch as secret and top-secret governmental networks. Data can beanonymized from company-to-cloud for enhanced learning at a high-leveldomain category beneficial to all users within that domain or across theentire cloud-based IVA platform.

The IVA may train machine learning models on diagrams, schematics, andmanuals, plus events, incidents, and human behavioral responses. Forexample, the IVA may learn a given user's daily routine, andautomatically provide certain information to that user at specific timesof the day. The IVA may have access to files for displaying on devicesand sending across a network. In some embodiments, the IVA platform maybe linked to similar IVA platforms at other companies in the same orsimilar field(s) to supplement and strengthen learning activities. Insome embodiments, the IVA platform may be communicatively connected toIOT and automation devices in home, industrial, and commercial settings.

Embodiments contained in the figures above may be used for other usecases related to industrial facilities and governmental agencies thatoperates energy, sewer, and water systems and may include other personaswithin the organization such as utility/facility director, operationsmanager, system operator, planning engineer, and field/maintenance crewor outside the organization such as outsourced third-party facilitiesand Operations and Maintenance (O&M) contractors. For each of thesepersonas, there is a typical pain point that can be efficientlyaddressed with the help of the industrial virtual assistant. Autility/facility director may have a pain point of an inability toreview up to date utility status and performance which could be solvedby an IVA by providing morning updates and weekly summaries. Anoperations manager may have a pain point because of difficultyprioritizing immediate tasks and response to events which could besolved by an incident response provided by an IVA. An operator may lackdata interoperability preventing optimal operation, solved again by anIVA that provides advanced warnings with root cause analysis. A planningengineer with limited ability to conduct analysis and gain insights foroptimal system upgrades can benefit from the information discoveryaspect of an IVA. A field or maintenance crew that lacks access to keyinformation needed for maintenance operations may use an IVA forinformation retrieval.

Embodiments may facilitate natural user interaction with data inindustrial facilities and governmental agencies operating energy, sewer,and water systems using an industrial virtual assistant with focuseddomain knowledge.

Embodiments may be comprised of: (a) a voice/text interface based onnatural processing engine for receiving queries and instructions from auser in the industrial facilities and governmental agencies operatingenergy, sewer, and water domain; (b) an asset management engine thatprovides information on assets mentioned by the user; (c) domainexpertise engine with the ability to provide the correct contextawareness to the user's query; (d) a data analytics engine that canimplement general purpose analytics on any data from SCADA, GIS systems,asset management systems, CMMS systems, and IoT sensors such as flowrate, level, pressure, chemical, temperature, vibration, power meters,and various databases; (e) data integration engine that allows use casedriven integration of multiple data streams from multiple sources; (f) aknowledge enhancement machine learning engine that can archiveconversations conducted between the user and the system and use theseconversation to improve the response in future conversation.

Embodiments may receive the query and/or instructions from the user andincorporates the natural language processing to support various usecases including incident response, advanced warning on potential assetfailure, asset information discovery and retrieval and periodic (daily,weekly) summary report.

Embodiments may incorporate the natural language processing enginecombined with the domain expert context awareness engine to classify thecorrect use case and to identify the data sources needed for theresponse to the user.

Embodiments may integrate the relevant data sources and performs therequired analytics on the data that will generate the required output asper the user query and/or instructions.

Embodiments may implement the domain expert context awareness andnatural processing engines to respond back to the user with the datadriven insights that will address the user query and/or instructions

Embodiments may receive limitless follow up queries and/or instructionsand then receives the query and/or instructions from the user andincorporates the natural language processing to support various usecases including incident response, advanced warning on potential assetfailure, asset information discovery and retrieval and periodic (daily,weekly) summary report. The industrial virtual assistant system andmethod incorporates the natural language processing engine combined withthe domain expert context awareness engine to classify the correct usecase and to identify the data sources needed for the response to theuser. The industrial virtual assistant system and method integrates therelevant data sources and performs the required analytics on the datathat will generate the required output as per the user query and/orinstructions. The industrial virtual assistant system and methodimplements the domain expert context awareness and natural processingengines to respond back to the user with the data driven insights thatwill address the user query and/or instructions

Embodiments may provide summary reports with all details of aconversation, including transcript, relevant data charts, visuals suchas photographs or drawings, and key insights.

Embodiments may archive the conversations with a user and uses a machinelearning engine for knowledge enhancement to improve response to user infuture conversations.

Embodiments may integrate enterprise-level services such as federatedservices to one or more users, LDAP architectures, and securityarchitectures.

For clarity and to provide a concrete example of an application of anIVA, this application uses the example of implementation of an IVA in anoil refinery. However, the system and method described herein are not inany way limited to any particular type of facility or field of use. TheIVA may be put to use in any complex system involving equipment,sensors, inventory, assets, and/or their control and/or asset managementsystems. Some non-limiting examples of applications to which the IVA maybe put include oil refineries, manufacturing facilities, warehousefacilities, building HVAC systems, refrigeration systems, and securitysystems and access controls for buildings, facilities, corporations,military bases, etc.

One or more different aspects may be described in the presentapplication. Further, for one or more of the aspects described herein,numerous alternative arrangements may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the aspects contained herein or the claims presentedherein in any way. One or more of the arrangements may be widelyapplicable to numerous aspects, as may be readily apparent from thedisclosure. In general, arrangements are described in sufficient detailto enable those skilled in the art to practice one or more of theaspects, and it should be appreciated that other arrangements may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularaspects. Particular features of one or more of the aspects describedherein may be described with reference to one or more particular aspectsor figures that form a part of the present disclosure, and in which areshown, by way of illustration, specific arrangements of one or more ofthe aspects. It should be appreciated, however, that such features arenot limited to usage in the one or more particular aspects or figureswith reference to which they are described. The present disclosure isneither a literal description of all arrangements of one or more of theaspects nor a listing of features of one or more of the aspects thatmust be present in all arrangements.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an aspect with several components in communication witheach other does not imply that all such components are required. To thecontrary, a variety of optional components may be described toillustrate a wide variety of possible aspects and in order to more fullyillustrate one or more aspects. Similarly, although process steps,method steps, algorithms or the like may be described in a sequentialorder, such processes, methods, and algorithms may generally beconfigured to work in alternate orders, unless specifically stated tothe contrary. In other words, any sequence or order of steps that may bedescribed in this patent application does not, in and of itself,indicate a requirement that the steps be performed in that order. Thesteps of described processes may be performed in any order practical.Further, some steps may be performed simultaneously despite beingdescribed or implied as occurring non-simultaneously (e.g., because onestep is described after the other step). Moreover, the illustration of aprocess by its depiction in a drawing does not imply that theillustrated process is exclusive of other variations and modificationsthereto, does not imply that the illustrated process or any of its stepsare necessary to one or more of the aspects, and does not imply that theillustrated process is preferred. Also, steps are generally describedonce per aspect, but this does not mean they must occur once, or thatthey may only occur once each time a process, method, or algorithm iscarried out or executed. Some steps may be omitted in some aspects orsome occurrences, or some steps may be executed more than once in agiven aspect or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other aspects need notinclude the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular aspects may include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of various aspects in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

Definitions

“Artificial intelligence” or “AI” as used herein means a computer systemor component that has been programmed in such a way that it mimics someaspect or aspects of cognitive functions that humans associate withhuman intelligence, such as learning, problem solving, anddecision-making. Examples of current AI technologies includeunderstanding human speech, competing successfully in strategic gamessuch as chess and Go, autonomous operation of vehicles, complexsimulations, and interpretation of complex data such as images andvideo.

A “database” or “data storage subsystem” (these terms may be consideredsubstantially synonymous), as used herein, is a system adapted for thelong-term storage, indexing, and retrieval of data, the retrievaltypically being via some sort of querying interface or language.“Database” may be used to refer to relational database managementsystems known in the art, but should not be considered to be limited tosuch systems. Many alternative database or data storage systemtechnologies have been, and indeed are being, introduced in the art,including but not limited to distributed non-relational data storagesystems such as Hadoop, column-oriented databases, in-memory databases,graph databases and the like. While various aspects may preferentiallyemploy one or another of the various data storage subsystems availablein the art (or available in the future), the invention should not beconstrued to be so limited, as any data storage architecture may be usedaccording to the aspects. Similarly, while in some cases one or moreparticular data storage needs are described as being satisfied byseparate components (for example, an expanded private capital marketsdatabase and a configuration database), these descriptions refer tofunctional uses of data storage systems and do not refer to theirphysical architecture. For instance, any group of data storage systemsof databases referred to herein may be included together in a singledatabase management system operating on a single machine, or they may beincluded in a single database management system operating on a clusterof machines as is known in the art. Similarly, any single database (suchas an expanded private capital markets database) may be implemented on asingle machine, on a set of machines using clustering technology, onseveral machines connected by one or more messaging systems known in theart, or in a master/slave arrangement common in the art. These examplesshould make clear that no particular architectural approaches todatabase management is preferred according to the invention, and choiceof data storage technology is at the discretion of each implementer,without departing from the scope of the invention as claimed.

“Machine learning” as used herein is an aspect of artificialintelligence in which the computer system or component can modify itsbehavior or understanding without being explicitly programmed to do so.Machine learning algorithms develop models of behavior or understandingbased on information fed to them as training sets and can modify thosemodels based on new incoming information.

“Neural network” as used herein means a computational model,architecture, or system made up of a number of simple, highlyinterconnected processing elements which process information by theirdynamic state response to external inputs and is thus able to “learn”information by recognizing patterns or trends. Neural networks, alsosometimes known as “artificial neural networks” are based on ourunderstanding of the structure and functions of biological neuralnetworks, such as the brains of mammals. A neural network is a frameworkfor application of machine learning algorithms.

“NLP” refers to natural language processing, which is a subset of AItasked with enabling machines to interact using natural languages. Thedomain of NLP also ensures that machines can process large amounts ofnatural language data and derive insights and information.

“NLU” refers to natural language understanding. NLU a subtopic of NLP,the main focus of natural language understanding is to make machinesinterpret the natural language, derive meaning, identify context, anddraw insights.

“NLI” refers to natural language interaction and refers to an overallinteraction experience wherein a user may use natural language tointeract with a system. According to the various embodiments herein, anNLI application such as a virtual assistant may utilize NLUfunctionality, and it should be appreciated that NLU is used to refer toan internal function and NLI is used herein to refer to an overallarrangement or experience.

A “language recognition rule” or “NLU rule” is a specific type ofcondition built up from language objects and used for capturing naturallanguage expressions. For example, a language recognition rule can beused to interpret or capture the intention of a user request.

As used herein, a “language object” means an abstract representation ofa logical unit of human linguistic expression that has meaning and issuitable for processing by automated systems such as virtual assistants.Language objects, in their simplest form, are represented as singlewords that represent a plurality of variants of a single common meaning,including inflectional variants and variants connected by synonymy. Thatis, generally a language object represents all variants and synonyms ofthe core word that represents it, and language objects may containgrammatical variants of words as well (such as verb tenses,contractions, and so forth).

Conceptual Architecture

FIG. 1 is a block diagram illustrating integration of a distributedindustrial virtual assistant platform into an industrial facility. Anyutility or industrial facility will typically have equipment to beoperated and a control center for operating the equipment. In smallerfacilities, these may be co-located, but in larger facilities (e.g.,utilities) the equipment may be located remotely from the controlcenter(s). A typical utility or industrial facility will have anoperations center 110 from which the equipment 140 is controlled. Forexample, in an oil refinery, the operations center 110 is typically aseparate building located on the site of the refinery. Facilitiesoperations staff (facilities manager, process control engineers, etc.)monitor and control the complex array of equipment 140 that refines thecrude oil into various petroleum products. The operations center 110will have a number of computer workstations 160 through which equipmentand processes may be monitored, and other computer equipment 170 such asservers, networking equipment, etc., to which the workstations 160 areconnected. Larger operations centers will have conference rooms andother special-purpose rooms 180 which may have telephones, videoconferencing equipment, audio-visual equipment, etc.

Smaller equipment 140 (e.g., valves, small pumps, etc.) will betypically operated directly, while larger, more complex equipment may beoperated via equipment control consoles 130 which may be located on theequipment or remotely. Processes involving multiple pieces of equipment140 are controlled by process control consoles 120, which are typicallylocated in the operation center 110 for the facility. Using processcontrol consoles 120, complex sequences of operations can be monitoredand implemented. All modern process control consoles 120 arecomputer-implemented, but some may contain displays and controls inpurpose-built panels or devices. In larger facilities and for largerprocesses, the process control consoles are typically software-based andoperated through workstations 160 that are remote from the equipment140. The facility and/or its operational staff may also have mobiledevices such as smart phones and tablet computers that allow for staffcommunications and/or act as process control consoles 120 or equipmentcontrol consoles 130.

Sensors 150 may be located throughout the facility, including on-boardsensors on equipment 140, sensors in equipment control consoles 130, andsensors unattached to specific equipment (e.g., ambient temperaturesensors).

The core of operational control of modern industrial facilities aresupervisory control and data acquisition (SCADA) systems. SCADA systemscomprise programmable logic controllers (PLCs) and remote terminal units(RTUs) which are microcomputers that are connected with the equipment140 and sensors 150 throughout the facility. Depending on theirconfiguration, the PLCs and RTUs acquire data from equipment, controlequipment, or both. The PLCs and RTUs are connected to SCADA softwarethat allows for monitoring of, and control of, the equipment, allowingfor sophisticated, computer-controlled processes to be implemented onthe equipment.

The distributed Industrial Virtual Assistant (IVA) platform (sometimescalled “IVA platform” herein) 200 is a software layer on top of theSCADA system (or integrated into it) which provide AI-based querying andcontrol of the SCADA system and its associated processes and equipment.Thus, as shown by the dotted lines in the diagram, the distributedIndustrial Virtual Assistant (IVA) platform is connected via the SCADAsystem to all of the components of the facility 110-180 and is capableof obtaining and analyzing information from all such components and,depending on configuration, controlling the processes and equipment ofthe facility.

FIG. 2 is a block diagram illustrating an exemplary system architecture200 for a distributed industrial virtual assistant platform. The systemarchitecture of this embodiment comprises a communications manager 300,an IVA manager 400, a domain expertise manager 500, an adaptation engine600, a data integration and analytics manager 220, a security module210, an incident manager 240, and a federation manager 230.

The communications manager 300 is responsible for data transfer betweenthe IVA platform and the facility's SCADA system and for facilitatinguser interactions via mobile devices and workstations. The IVA manager400 is responsible for generating and instantiating device-specific anduser-specific industrial virtual assistant instances based on userprofiles and domain expertise, and for storing and migrating IVAinstances across users, devices, and workstations. The domain expertisemanager 500 is responsible for managing domain-specific knowledge andfacilities asset information, providing such knowledge and informationin response to queries, and for coordinating feedback about suchknowledge and information with the adaptation engine. The adaptationengine 600 is responsible for evaluating new knowledge and informationand for coordinating changes to the domain-specific knowledge andfacilities asset information with the domain expertise manager. Thesecomponents will each be described in further detail in separate sectionsbelow.

The security module 210 is responsible for maintaining security ofinformation passed to workstations and, in particular, to mobiledevices. The security module ensures encryption of communications withuser devices to prevent unintentional disclosure of sensitiveinformation and to prevent unauthorized access to systems. In someconfigurations, the security module 210 may establish virtual privatenetworks (VPNs) to user devices or use other forms of securecommunications.

The incident manager 240 is responsible for receiving rules andthresholds from the data integration and analytics manager, monitoringof equipment and sensor data, and notifying of incidents involving ruleadherence or violation. The federation manager 230 is responsible formanagement of the various databases of the platform (i.e., thedomain-specific knowledge databases 520, user profile and devicedatabases 440, IVA instance databases 450, etc.) as a distributeddatabase management system accessible by other components of theplatform. A federated database system is more scalable thannon-federated database system, and can be used to support virtualizationsuch as containerization of the IVA instances generated by the IVAmanager 400.

FIG. 3 is a block diagram illustrating an exemplary architecture for thecommunication manager aspect of the exemplary a distributed industrialvirtual assistant platform. The communications manager 300 isresponsible for data transfer between the IVA platform and thefacility's SCADA system and for facilitating user interactions viamobile devices and workstations. The communications manager 300comprises a natural language processing engine 310, a proxy anddistribution manager 320 and a SCADA integration module 330.

The natural language processing (NLP) engine 310 receives usercommunications in the form of voice and/or text. The NLP engine 310 usesspeech recognition algorithms to convert any voice communications totext, and then uses domain-specific NLP machine learning algorithms toparse the text to understand the context of the communication (e.g.query for information, a control or configuration command, etc.) and toidentify the assets, equipment, or resources to which the communicationis directed (e.g., pump X, valve Y, etc.). The domain-specific NLPmachine learning algorithms of the NLP engine 310 are trained usingsample user communications in conjunction with the domain-specificknowledge database(s) 520 and facilities database(s) 540.

The proxy and distribution manager 320 is responsible for establishingand maintaining communications with users via mobile devices andworkstations. The SCADA integration module is configured to receive andtransmit data and control commands between the facility's SCADA systemand the IVA platform 200.

FIG. 4 is a block diagram illustrating an exemplary architecture for theIVA manager aspect of the exemplary a distributed industrial virtualassistant platform. The IVA manager 400 is responsible for generatingand instantiating device-specific and user-specific industrial virtualassistant instances based on user profiles and domain expertise, and forstoring and migrating IVA instances across users, devices, andworkstations. The IVA manager 400 comprises an IVA instance generator410, an IVA instance database 450, a user profiles and devices database440, a user profiles and devices manager 430, a session and migrationmanager 420, and a configuration manager 460.

The IVA instance generator 410 generates user-specific, device-specific,and domain-specific instances of an IVA for specific users, groups, ordevices by selecting domain-specific knowledge database that correspondsto information in the user profiles and devices database 440. Forexample, an electrical engineer working at a facility is not likely tobe interested in fluid controls domain knowledge such as flow rates andpump pressures at the facility, so the information selected for the IVAinstance for that engineer can be limited to the domain of electricalengineering (i.e., currents, voltages, wiring and connections, controlsystems such as proportional-integral-derivative (PID) controller,etc.). Further, this engineer may use only two devices at the facility,a smartphone and a workstation, which information will be contained inthe user profiles and devices database 440, and may be used to install asmaller version (e.g., with fewer domains of knowledge or limitedcapabilities) in the smartphone, with a larger version installed on theengineer's more powerful desktop workstation. Thus, the IVA instancegenerator can generate IVA instances for each user that are customizedto that user's role at the facility and that user's devices.

The IVA instance database 450 is used to store and manage the variousinstances of IVAs created for the operational staff at the facility. Thesession and migration manager 420 keeps track of which IVAs are activefor which users at a given time, and can migrate an IVA instance todifferent devices, users, or groups, as necessary to maintaincontinuity. For example, in the case of the electrical engineerdiscussed above, if the engineer is working onsite with equipmentlocated at a remote location and accesses the IVA instance on hissmartphone, when he or she gets back to the office and starts working onhis or her workstation, the interactions with the smartphone's IVAinstance are migrated to the IVA instance on the engineer's workstationto maintain continuity of operations.

The user profiles and devices database 440 stores information aboutoperational staff at the facility relevant to generation and use of IVAssuch as the user's name, occupation, operational role at the facility,office location, access privileges and authorities, assigned computersand other devices, relevant knowledge domains (e.g., piping, electrical,construction, process controls, etc.). The user profiles and devicesmanager 430 can be used by administrative staff to update and manage theinformation contained in the user profiles and devices database.

The configuration manager 460 can be used by appropriate operationalstaff to select which information contained in the user profiles anddevices database 440 should be used by the IVA manager 400 to generateIVA instances for a given user. In the case of the electrical engineermentioned above, the configuration manager 460 can be used to manuallyforce inclusion of the fluid control domain knowledge that wouldotherwise be excluded by the IVA instance generator's 410 automated IVAinstance generation for that engineer.

FIG. 5 is a block diagram illustrating an exemplary architecture for thedomain expertise manager aspect of the exemplary a distributedindustrial virtual assistant platform. The domain expertise manager 500is responsible for managing domain-specific knowledge and facilitiesasset information, providing such knowledge and information in responseto queries, and for coordinating feedback about such knowledge andinformation with the adaptation engine. The domain expertise manager 500comprises a query handler 510, a domain-specific knowledge database 520,a facilities database 540, a domain cross-over manager 530, and a domainconfiguration manager 550.

The query handler 510 receives queries from the IVA manager 400 fordomain-specific or facility-specific information related to a queryprocessed by the NLP engine 310, retrieves the requested informationfrom the domain-specific knowledge database 520 or the facilitiesdatabase 540, as appropriate, and returns the requested information tothe IVA manager 400.

The domain-specific knowledge database 520 comprises informationorganized into domains of knowledge or expertise such as construction,electrical engineering, civil engineering, mechanical engineering,processes and process controls, piping, equipment, etc. Each domaincontains information and rules about that domain of knowledge such asterminology, formulas, objects, relationships, interactions, and otherinformation that experts in that field would be expected to know or befamiliar with. The information and rules about each domain can bemodified over time by training and re-training of machine learningalgorithms using voice, text, and visual interactions with users and/orequipment over a period of time to determine which actions and behaviorsgovern incidents. The rules may include checking on one or multipleexceptions, existence of various conditions, simple statistical checkson various data sets and even including static meta information aboutthe assets involved. From the information and rules, IVAs can beconstructed that are domain-specific, and can be further customized tobe user-specific and/or device-specific. The information and rules usedto generate IVAs may change over time as the system learns from userinteractions, exceptions, and incidents. The domain-specific knowledgedatabase 520 also stores learning about complex incident behaviors(e.g., in industrial/utility processes). From these complex incidentbehaviors, IVAs can be generated to connect data from various systemsand over time to anticipate problems and prescribe actions to users. Thedomain-specific knowledge database 520 can contain information and rulesabout vendor-specific equipment allowing companies to expertly craftinteractions between an IVA instance, a given user, and relevantfacility assets, equipment, or processes.

The information and rules contained in the domain-specific knowledgedatabase 520 may be used to create and establish thresholds and/ortriggers which can be used to monitor data from certain pieces ofequipment or from certain processes. Thresholds and triggers may bequite complex, requiring multiple steps or multiple points of data to bemet, but two simple examples are establishment of a set point or mayestablish a range of operation values. When a threshold or trigger ismet, a notification of the occurrence of the threshold event or triggermay be sent to the user.

The facilities database 540 contains facility-specific information aboutequipment, components, sensors, and processes of the facility, and isconfigured to store current and historical data about the current statusof the facility (e.g., equipment operational statuses, sensor data, flowcontrols, current processes in operation, etc.) such that historicaldata can be accessed by the IVA platform. The facilities database 540may contain information not only about the facilities assets (e.g., tankcapacities, pipe capacities and flow rates, pump pressures, etc.), butalso about processes within the facility (e.g., flow rates are notallowed to exceed a certain rate during certain operations, increasingprocessing volumes of a given operation will reduce efficiency by acertain percentage, etc.). The facilities database 540 may be updated bycontinual re-training of machine learning algorithms based on exceptionsand/or incidents identified during the interactions. As one example, ifthe facilities database 540 indicates that a particular pump should beoperating at a minimum pressure of 100 psi, but an operator determinesthat a given process works better if the pump is operated at 90 psi, thesystem will learn that the preferred operation for that pump at thatfacility is 90 psi, not 100 psi.

The domain cross-over manager 530 identifies and coordinates informationthat corresponds to multiple domains. For example, information about thenominal operating pressure of a given type of pump may be relevant tomultiple related domains such as pumps, piping, process controls,mechanical engineering, and other domains. Therefore, if learning aboutthat given type of pump occurs in one domain, domain cross-over manager530 ensures that that learning is propagated to other related domains toensure consistency in analysis and responses by IVAs generated for thosedomains.

Domain configuration manager 550 may be used by appropriate operationalstaff to make manual changes to the databases 520, 540 or operation ofthe query handler 510 or domain cross-over manager.

FIG. 6 is a block diagram illustrating an exemplary architecture for theadaptation manager aspect of the exemplary distributed industrialvirtual assistant platform. Adaptation engine 600 is responsible forevaluating new knowledge and information and for coordinating changes tothe domain-specific knowledge and facilities asset information with thedomain expertise manager. Adaptation manager 600 is the adaptivelearning component of the platform and comprises an exception data andincident report handler 601, a domain machine learning algorithm (MLA)602, a context machine learning algorithm (MLA) 604, a domain learningdatabase 603, and a context learning database 605.

Exception data and incident report handler 601 receives notification ofexceptions from a given IVA instance, forwards the exception to thedomain MLA 602 and forward the context in which the exception occurredto the context MLA 604. As an example, an IVA instance may tell a userthat the nominal (name plate) operating pressure of a given pump is 100psi, but in response the user tells the IVA that the pump operatesbetter at 90 psi. The user's response is an exception and the context ofthe response is that a certain engineer on a certain day at a certaintime using a certain device (and possibly under certain circumstancessuch as during a certain process) said that the pump operates better atsomething other than the nominal pressure.

The domain machine learning algorithm (MLA) 602 receives exceptions andprocesses them to learn new information about the subject of theexception (in the example above, about pump operating pressures for agiven pump), which information may be forwarded to the domain expertisemanager for direct incorporation, and which may be generalized by thedomain MLA to similar types of pumps if similar exceptions are receivedover time. This learning information is stored in the domain learningdatabase 603. More information is described below about machine learningmodels that may be used by the IVA platform.

The context machine learning algorithm (MLA) 604 receives exceptions andprocesses them to learn new information about the context in which anexception occurred so as to generalize that learning to other similarcontexts. For example, if the engineer makes the same exception aboutlowering the operating pressure of the pump to 90 psi at a particulartime each day, the system can learn that the pressure for that pumpshould be lowered each day at that time, and can either prompt theengineer to do so, or ask the engineer if that procedure should beimplemented automatically each day. Likewise, if a different engineer isworking on a given day, the system can notify the different engineerthat the other engineer usually lowers the pump pressure at that timeeach day. This learning information can be stored in the contextlearning database 604, and the learning outcomes as described above maybe forwarded to the domain expertise manager 500 for incorporation intothe domain-specific knowledge database 520. More information isdescribed below about machine learning models that may be used by theIVA platform.

Detailed Description of Exemplary Aspects

FIG. 7 is an illustration showing a simplified layout of an oil refineryas an example of an industrial facility into which a distributedindustrial virtual assistant platform may be integrated. In thisexample, the oil refinery comprises a refining system 720 comprisingcrude oil tanks 721 in which crude oil is stored before processing, acrude oil heater 722 which heats the crude oil for separation intofractions by the distillation tower 723, a distillation tower 723 inwhich the heated crude oil is separated into heavier and lighterfractions by drawing off and condensing the gasses from the crude oil atdifferent heights within the tower, a catalytic cracking unit 725 whichuses heat, pressure, catalysts, and sometime hydrogen to crack heavyhydrocarbon molecules into lighter ones, a sulfur recovery processor 724which removes hydrogen sulfide from the products and converts them intonon-toxic elemental sulfur, a waste treatment unit 727 a and waste tanks727 b which receive wastewater and allow solids and contaminants tosettle out, product tanks 726 which store the refined petroleumproducts. The entire process is controlled from operations center 710staffed by engineers and other knowledgeable professionals. This diagramshows the piping network 730 between each of the units of the refiningsystem 720 with flow controllers 731 and flow meters 732 along each ofthe pipes, level sensors 728 on the tanks 721, 726, equipmentcontrollers 729 associated with each processing unit, and a processcontroller 711 located at the operations center 711, along with a wiringnetwork 740 connecting each flow controller, flow meter, equipmentcontroller, and level sensor to the operations center 710. The wiringnetwork plus its connections constitutes the hardware of the refinery'sSCADA system, all managed by software at the operations center 710.Operations staff may use workstations at the operations center 710, ormay use mobile devices 712 for system control while outside of theoperations center 710. Note that this process is necessarily simplifiedfor clarity and shows exemplary connections rather than all possibleconnections.

The IVA platform 200 is a software layer on top of the SCADA system (orintegrated into it) which provide AI-based querying and control of theSCADA system and its associated processes and equipment. Thus, the IVAplatform 200 is connected via the SCADA system to all of the componentsof the refining system 720 and is capable of obtaining and analyzinginformation from all such components and, depending on configuration,controlling the processes and equipment of the facility.

FIG. 8 is a method diagram illustrating an exemplary process forre-training of domain and context machine learning algorithms based onuser interactions. Initially, a domain machine learning algorithm (MLA)is trained using pre-existing and/or curated domain databases 801 whichcontain domain expertise knowledge such as construction industryterminology, piping terminology, electrical terminology, processcontrols terminology, and rules and other expert knowledge relevant toeach such domain. Likewise, a context MLA is trained usingfacility-specific and/or process-specific databases 802 containinginformation such as equipment specifications, component specifications,process steps and details, locations, people, etc. When a user query isreceived by an IVA instance 803 the IVA performs natural languageprocessing to convert any voice communications to text, and to parse thetext of the communication into words and phrases 804. The parsed text isused to query the domain expertise manager to confirm the meaning of thewords and phrases in context 805 and them to identify equipment,processes, components, sensors, of the facility to which the user queryis directed 806. Data is retrieved for the identified items 807, and aresponse to the query is provided to the user 808. This process isrepeated for any follow-up queries the user may have for additionalinformation or more specific information, and the follow-up queries areforwarded to the context MLA for re-training based on the context of thefollow-up queries (e.g., if the user asked for “today's” pump pressurefor pump X and got the current pressure, he or she may follow up to askabout the average pressure for the day, which context will beincorporated by the context MLA to better handle future such requests).In the event that the user provides an exception to the response 809(e.g., “the pump pressure for that pump should be 90 psi, not 100 psi”),the exception details are provided to the domain MLA for re-trainingbased on the new information 810 and the context for the exception isprovided to the context MLA for retraining 811 (e.g., this user on thisday at this time requested a lower pump pressure for this pump).

FIGS. 9A & 9B are flow diagrams illustrating an exemplary operationaluse of a distributed industrial virtual assistant platform to assist inthe operation of an industrial facility. Here, messaging is shownbetween a user 910 and the system 930, along sequences of operationsoccurring either at the IVA instance or the IVA platform.

The IVA manager of the IVA platform generates a user-specific anddevice-specific IVA 931 for the facilities manager (user) of a facilityand uploads the generated IVA instance to the facilities manager'sdevice 911. Using the IVA, the user asks for the fluid pressure in pumpX 912. This query is received by the communications manager of the IVA932 which performs speech-to-text conversion and identifies the keyterms “fluid pressure” and “pump X” using its natural languageprocessor. The parsed text is sent to the domain expert manager toidentify relevant facility assets, processes, and rules 933. The dataintegration and analysis manager retrieves the requested data and/orapplies the appropriate rules 934 and sends the data to thecommunications manager for generation of a response using its NLP engine935. The user receives the response “the fluid pressure in pump X iscurrently 110 psi” 913, to which the user replies with a follow-up query“tell me if the pressures in pump X fall outside of normal parameters”914. The same process of parsing the query and identifying assets asabove is performed 936, 937, but note that the term “normal parameters”must be determined by the domain expertise manager from thedomain-specific knowledge database. Once the normal parameters of thatpump are determined, thresholds can be set and sent to the dataintegration and analysis manager for monitoring 938, and a naturallanguage response is generated 939 confirming that the IVA will notifythe user if the pressures in pump X fall outside normal parameters 915.Upon triggering of a rule or threshold at the data integration andanalysis manager, another natural language response is generated 941stating that the minimum pressure of pump X has fallen below the normalthreshold of 100 psi 916, to which the user responds by stating theexception “the minimum pressure of Pump X should be 90 psi” 917. Thesame process of parsing the query as above is performed 942, but thistime an exception is noted and an exception report is sent to theadaptation engine 943. The platform IVA manager updates theuser-specific and device-specific IVA to reflect the new information inthe exception 944 and the updates are uploaded to the facilities managerdevice IVA 918, along with a natural language generated response 945confirming that the minimum pressure threshold for Pump X has been setto 90 psi, and asking whether this new rule should be applied to allpumps of type X 919. A positive response by the user 920 and naturallanguage recognition by the IVA communications manager 946 sets inmotion an update process which identifies all affected assets 947,retrains the machine learning algorithms in using both the new domaininformation (90 psi for pumps of type X instead of 100 psi) and the newcontext information (from the facilities manager using a particulardevice at a particular time on a particular day, etc.) 948, updates thedomain-specific knowledge databases based on the retraining 949, andfinally updates the user-specific and device-specific IVA 950 anduploads it to the facility manager's device 921.

FIG. 10 is a block diagram illustrating an exemplary conceptual mappingarchitecture for dynamic concept learning. This flow diagram shows anexemplary method for training an industrial virtual assistant. A userinterface is provided that allows for users to provide textual listingof domain concepts 1001. A domain concept 1001 may be classified intovarious entities such as: measured entities 1011, calculated entities1012, spatial entities 1013, and exception criteria 1014.

From these entities, a concept map 1020 may be developed wherein a linksmeasured entities 1011 with corresponding measured concepts 1021.Measured data may be stored in a measured data database 1041. User mayuse a visual interface or voice commands to establish this linking.

The user may define a time-series processing-based criteria 1012 andlink it with a calculated concept 1022, using a visual interface orvoice commands to provide the criteria definition.

The user may link elements on a GIS map (individual or a spatiallygrouped) 1013 with spatial concepts 1023 previously defined, using avisual interface or voice commands to establish this linking.

The user may define an exception concept 1014 whereby the exceptioncriteria 1024 provided by the user can be executed on a measured concept1021 or a calculated concept 1022, using a visual interface or voicecommands to provide the criteria definition.

A natural language processing engine—which may also comprise naturallanguage understanding (NLU)—may be used to extract the “intent” and“entity” from user's speech/text describing a calculated concept 1022and its definition. For example: a user provides a definition of acalculated concept 1022, as speech or text or via a visual userinterface, for a concept called “Nightline”: which may be defined as MinNight Flow on All Flow Sensors. Then a user provides the followingdefinition of an exception concept 1024, as speech or text or via avisual user interface, for a concept called “Critical Nightline”:Critical Nightline if Nightline is on an uptrend. Lastly, a userprovides the following definition of a spatial concept 1023, as speechor text or via a visual user interface, for a concept called “EasternZone”: All sensors located within this polygon are part of Eastern Zone.These are passed to the NLP engine to extract the intent and entities.From example above, Intent: Min Night Flow and Entities: All, FlowSensors.

Intents may be mapped to generic or domain specific calculationlibraries, generic or domain specific geographic definitions, generic ordomain specific exception definitions, or any combination thereof.

Supported generic or domain specific calculation libraries comprisegeneral calculations which include but are not limited to maximum,minimum, average, daily/nightly minimum, daily/nightly maximum,daily/nightly average, rate of change, delta, uptrend, downtrend, trend.Examples of domain specific calculations include but are not limited to(water domain in this case): Min Night, Daily Consumption, DemandMetered Area (DMA) InFlow, Water Balance etc.

Supported generic or domain specific geographic definitions comprisegeneral geographic definitions which include but are not limited to partof (polygon, circle, square), at (location (defined by a point on ageographic map e.g., latitude/longitude)), group of locations, postcode.Examples of domain specific geographic definitions include but are notlimited to (water domain in this case): District Metered Area (DMA),Demand Zone (DZ), Network

Supported generic or domain specific exception definitions comprisegeneral exception definitions include but are not limited toless/greater than, is equal, is uptrend, is downtrend, rate of changeequal or greater/less than, no data, no change in data. Examples ofdomain specific Exception definitions include but are not limited to(water domain in this case): Noisy Flow, Bad Acoustic, No Solar Charging

Entities may be mapped to measured concepts 1021 or other calculatedconcepts 1008 to determine target of the intent, and then registered asa new concept in the concepts map 131.

For each registered concept in the concepts map 1020, algorithmsexecuted by various engines 1032-1034 such as following are provided tosatisfy the concept's intent. A calculation engine 1032 executescalculated concept intents on entities to produce a new calculation andstore most recent values in a calculated data cache 1042 for quickretrieval. A geo definition engine 1033 executes spatial concept intentson entities to produce definitions of new geographical concepts andstore any new geographical concept definitions in a geo database (GeoDB) 1043. An exception detection engine 1034 executes exception conceptintents on entities to detect an exception and stores the most recentexceptions in an exceptions cache 1044 for quick retrieval.

FIG. 11 is a flow diagram illustrating a dynamic concept learningprocess. Upon text or voice command from user 1101, the system uses anatural language processing engine 1102 to parse the query to identifythe intent and entities in the user's query. For newly identifiedconcepts and entities, the parsed query is sent to a new conceptprocessing system 1110 comprising a calculation engine 1111, ageographical definition engine 1112, and an exception detection engine1113. The calculation engine 1111 executes calculates the concept“intent on entities” to produce a new calculation applying theidentified intent to the identified entities in the parsed query, andstore most recent values in a calculated data cache 1121 for quickretrieval. The geographical definition engine 1112 produces acorresponding geospatial definition of “intent on entities” and storesthe geographical concept definition in geospatial cache 1122. Theexception detector 1113 analyzes the calculated “intent on entities” forcorrespondence with an existing concepts map 1130 and generates anexception if that calculated “intent on entities” is not alreadycontained in the concepts map 1123 or conflicts with data already in theconcepts map 1130. Generated exceptions are stored in an exceptionscache 1123 for quick retrieval.

FIG. 12 is a flow diagram illustrating the workflow from a plurality ofconversations 1200 a-n to a generation of an incidents network. A voiceand text based Industrial Virtual Assistant (IVA) may be trained usingvoice, text, and visual interactions over a period of time on variousrules that govern incident detection criterion. The rules may includechecking on one or multiple exceptions, existence of various conditions,simple statistical checks on various data sets and even including staticmeta information about the assets involved.

For each incident type, the IVA may be provided different sets of rulesover time. In the backend, the IVA dynamically builds a convergence mapthat guides it to apply the best set of rules and make a highly accurateincident detection and classification for similar future incidents.

This dynamic learning also eliminates the laborious work by domainexperts and operators of having to check various datasets and exceptionsmanually each time. The IVA, after selecting the best set of rules,automatically does those checks and brings actionable insights to theusers, saving significant amount of time during time critical incidentresponse scenarios.

According to another learning aspect of various embodiments, a userinterface may be provided for a user to ask questions and receiveresponses in regard to measured data (such as data from sensing systems,asset information etc.), calculated data (data derived from sensor datae.g., maximum daily pressure derived from pressure data), spatial (datarepresenting a geographic entity such as an area/zone) and exceptions(exceptions detected on measured and/or calculated data e.g. abnormalpressure detected on pressure data). A user may explicitly command theIVA to capture criteria definition of an incident from on-goingconversation. A visual user interface may be provided allowing a user toselect one or multiple parts of the on-going conversation to be includedin the incident criteria. Inputs may be taken from the conversation andused to generate a structured representation of the incident criterioncalled incident model 1202-1204 via an incident modeling system 1201.

A data model named incident map 1206-1208 may be provided to representvarious criteria rules related to different incidents and on which partsof the system (sensors, geographic locations etc.) these rules apply.The incident models 1202-1204 are ingested into an incident map1206-1208 via an incident mapping system 1205. A machine learning method1209 generates an incident network 1210. Supervised or unsupervisedmachine learning may be used. Some machine learning methods may bepreferred such as an artificial neural network, deep learning network,or a decision tree network which may be used to learn from multiplecriteria rules related to the same incident. Each time an incident isdetected using an existing criterion or a new criterion is provided bythe user, the machine learning method is employed to update theunderlying machine learning model using the most up-to-date incidentmap.

FIG. 13 is a flow diagram illustrating the workflow for using learningsfrom past user conversations to classify new incidents and provide aricher incident classification experience. An exception detection engine1301 detects a measured and/or calculated data exception, e.g., AbnormalPressure detected on Pressure data. Once a detection occurs, anexception detection engine 1301 is triggered to check the incident maps1302 to determine the best possible incident, named candidate incidentX, which this exception may point to. The rules provided in the incidentmap of the candidate incident 1303 are followed and tested 1304. Therules may involve checking for more exceptions and/or retrieving othermeta data. An incident Classifier 1306 is provided that receives theoutputs of rule tests 1305 and feeds the output 1305 into the incidentsnetwork 1210 to determine the incident class 1307. A visualization isprovided where the detected Incident Class 1307 along with details onthe executed rules are relayed to the user.

A tracking daemon may be employed that periodically checks on theparameters in the incident map 1303 and looks if any of the exceptionconditions specified in the incident map 1303 are satisfied. If so,occurrence of such additional exceptions as part of the same incidentare notified to the user on a mobile/desktop interactive visualization.A summary of the incident is generated and is presented on themobile/desktop interface. This summary includes and is not limited toincident type, time, location map; exceptions included in the incidentalong with the time series and geographical information; and status ofincident in terms of being active, under investigation, or closed.

FIG. 14 is a flow diagram illustrating how an industrial virtualassistant learns from unstructured information and adds that to anincident map for future retrieval. Following from FIG. 13 , avisualization interface may be used to capture additional situationsobserved during the incident that may or may not be directly captured byautomated and measured data. Such inputs may include on-the-groundobservations such as “traffic disruption,” “additional servicesdisruption,” “foul smell or taste” etc. Such inputs may be provided viastructured text or speech. Inputs may be taken from the conversation andused to generate a structured representation of the incident criteriacalled incident models 1405 via an incident modeling system 1401. A datamodel named incident map 1503 may be provided to represent variouscriteria rules related to different incidents and on which parts of thesystem (sensors, geographic locations, etc.) these rules apply. Theincident models 1405 are ingested into an incident map 1503 via anincident mapping system modeling system 1401. A machine learning method1409 generates an incident network 1410. Supervised or unsupervisedmachine learning may be used. Some machine learning methods may bepreferred such as an artificial neural network, deep learning network,or a decision tree network which may be used to learn from multiplecriteria rules related to the same incident. Each time an incident isdetected using an existing criterion or a new criterion is provided bythe user, the machine learning method is employed to update theunderlying machine learning model using the most up-to-date incident map1503.

This computer method passes these additional inputs to an incidentmapping system 1405 to be captured in the incident network 1410.

FIG. 15 is a block diagram illustrating an exemplary backendarchitecture for a distributed industrial virtual assistant platform200. The system output is an adaptive industrial virtual assistant (IVA)that defines domain specific concepts and contexts and can easilyextract the relevant insights from the backend databases. For example, auser who may be a facility manager/engineer might ask (by voice or text)the industrial virtual assistant that is installed on his or her mobiledevice what the fluid pressure is within in a certain pumping stationwhich is at a certain geographical location and at a specific time. Inthis case, the industrial virtual assistant has the relevant domainexpertise to understand the correct meaning of the technical terms:“fluid pressure in the pumping station.” And with this understanding,the virtual assistant can find the relevant asset ID in the GIS data,locate the relevant fluid pressure sensor ID in a SCADA system, andextract the relevant fluid pressure data at the requested geographicallocation and at the required time point. Thereafter, the industrialvirtual assistant will respond to the user (voice or text) with thefollowing answer: “the fluid pressure in pump x in location y and attime z is w psi.”

The industrial virtual assistant navigates back and forth through thecomponents spanning from an initial data layer 1540 to avoice/text-based user interface 1510. The data layer comprises at leastuse case driven data integration 1541 which directly informs theactionable information discovery 1531 and the knowledge enhancement 1532of the domain knowledge layer 1530. An insights layer 1520 providescontext awareness delivery 1521, an asset/equipment/sensor dataprovision service 1522, and a data analytics and alerts provisionservice 1523. User's may make queries from mobile devices 1511,workstations, within specific rooms or buildings (e.g., command center1512), and other specific devices and areas.

FIG. 16 is a diagram showing integration of exemplary knowledge sourcesof an industrial facility by a robotic process automater. Whileknowledge sources for an industrial facility are many, one way toclassify them is in terms of human inputs 1610, documentation 1620, andoperational data 1630. Human inputs 1610 to robotic process automater1700 are any changes, updates, or interactions with equipment, systems,or processes of a facility, or any interactions with an IVA platform 200associated with such equipment, systems, or processes including but notlimited to queries, comments, changes, updates, or notations aboutequipment, systems, or processes of the facility. Documentation 1620inputs to robotic process automater 1700 are any form of documentationabout equipment, systems, or processes of a facility including but notlimited to equipment manuals, user manuals, process manuals, operationalinstructions, and other facilities documentation such as that describedfor facilities database 540. Operational data 1630 for inputs to roboticprocess automater 1700 are any real-time, near-real-time, or stored datafrom equipment, sensors, logs, or other data-producing devicesassociated with a facility. Taken together, these three broadclassifications of data provide robotic process automater 1700 withinformation about expected behavior of equipment, systems, or processesin the form of documentation 1620, actual behavior of equipment,systems, or processes in the form of operational data 1630, andcorrections to, or modifications of, equipment, systems, or processes inthe form of human inputs 1610, which may then be incorporated into aknowledge graph showing the relationships between expectations, actualbehaviors, and corrections or modifications, and machine learningalgorithms may be applied to the knowledge graph to make predictionsabout future behaviors of equipment, systems, or processes in order toinform human operators or perform automated control of such equipment,systems, or processes.

FIG. 17 is an exemplary system architecture for a robotic processautomater aspect of an industrial virtual assistant. Robotic processautomater (RPA) 1700 allows operational staff at industrial facilitiesto have natural language conversations to obtain information about, andto control operations of, industrial facilities, and which automatescertain processes based in part on those natural language conversations.Robotic process automater (RPA) 1700 ingests information from humaninputs 1610, documentation 1620, and operational data 1630 from afacility, organizes that information into a knowledge graph containingcomprehensive facility information, and applies machine learningalgorithms to the knowledge graph to provide natural language responsesto human queries and to automate certain processes of the facility.Depending on configuration, robotic process automater (RPA) 1700 mayeither operate independently from Industrial Virtual Assistant (IVA)platform 200 as a stand-alone system, or may be a component or aspect ofIndustrial Virtual Assistant (IVA) platform 200. In this exemplaryarchitecture, robotic process automater (RPA) 1700 is shown as astand-alone system comprising a conversational AI engine 1710, a queryhandler 1720, a context recognition machine learning algorithm 1730, aknowledge graph 1740, a document ingester 1750, a domain machinelearning algorithm 1760 and a SCADA integration module 1770.

Conversational AI engine 1710 has functionality similar to naturallanguage processing engine 310 described in other embodiments.Conversational AI engine 1710 receives user communications in the formof voice and/or text. Conversational AI engine 1710 uses speechrecognition algorithms to convert any voice communications to text, andthen uses domain-specific NLP machine learning algorithms to parse thetext to understand the context of the communication (e.g. query forinformation, a control or configuration command, etc.) and to identifythe assets, equipment, or resources to which the communication isdirected (e.g., pump X, valve Y, etc.). Conversational AI engine 1710may comprise domain-specific NLP machine learning algorithms that may betrained using sample user communications or customer-specific taxonomiesin conjunction with the domain-specific and facility-specificinformation (for example, as might be contained in domain-specificknowledge database(s) 520, facilities database(s) 540) and/or knowledgegraph 1740).

Query handler 1720 has functionality similar to query handler 510described in other embodiments. Query handler 1720 receivesdomain-specific or facility-specific queries or commands from theconversational AI engine 1710 or domain-specific or facility-specificautomated queries or commands from domain MLA 1760, retrieves relevantinformation from knowledge graph 1740 (for example, if a query containsthe word Pump A, query handler 1720 may retrieve the specifications oridentifier for Pump A from knowledge graph 1720), and passes the queryand retrieved information to context recognition machine learningalgorithm 1730.

Context recognition machine learning algorithm 1730 has functionalitysimilar to context machine learning algorithm(s) 604 of otherembodiments. Context recognition machine learning algorithm 1730receives queries and processes them to learn new information about thecontext in which a query was made so as to facilitate generalization ofthat learning to other similar contexts. For example, if a facilitiesengineer makes a query about lowering the operating pressure of a givenpump to 90 psi at a particular time each day, the system can learn thatthe pressure for that pump should be lowered each day at that time, andcan either prompt the engineer to do so, ask the engineer if thatprocedure should be implemented automatically each day, or automaticallyimplement that process once a threshold level of certainty about thatcontext has been reached. Likewise, if a different engineer is workingon a given day, the system can notify the different engineer that theother engineer usually lowers the pump pressure at that time each day.This learning information can be stored in a database (e.g., contextlearning database 604), and the learning outcomes as described above maybe stored or forwarded to other relevant system components (e.g., forexample, forwarded to domain expertise manager 500 for incorporationinto the domain-specific knowledge database 520). More information isdescribed above about machine learning models that may be used by theIVA platform.

Knowledge graph 1740 is a database in graph form that stores complexrelationships between things and events. A knowledge graph is comprisedof nodes and edges, wherein the nodes represent things (any type ofthing, whether tangible or not, including but not limited to concepts,entities, and events) and the edges represent relationships between thethings (including but not limited to inferred relationships or insightsderived from the outputs of machine learning algorithms). Both nodes andedges of a knowledge graph may contain additional data about the things(for example, that the pump represented by the node operates as 15 psior that the relationship between nodes is of a certain character) andmay further contain additional information about the node or edge itself(often called metadata such as the number of edges connected to a givennode). Knowledge graphs thus represent a scalable, comprehensive set ofknowledge about a certain topic such as a facility and its operations.As knowledge graphs are a type of graph, any graph-based analyses can beperformed on them (e.g., shortest path analyses, least cost analyses,etc.). Consequently, knowledge graphs are ideal forms of data storagefor processing by machine learning algorithms.

Domain machine learning algorithm 1760 has functionality similar todomain machine learning algorithm(s) 602 of other embodiments. Domainmachine learning algorithm 1760 receives information of the three typesdescribed above (human inputs 1610, documentation 1620, and operationaldata 1630) and processes them to learn new information and obtaininsights about any aspect of a facility or its operations. Domainmachine learning algorithm 1760 incorporates the received data and anynew information or insights into knowledge graph 1740 to provide acomprehensive set of knowledge about the facility. Information andinsights may be generalized by the domain MLA to similar types ofequipment or similar contexts within the facility, allowing automatedpredictions and recommendations about facility equipment and operations.More information is described above about machine learning models thatmay be used by the IVA platform. In addition, through combining threepieces of knowledge, ML also helps to identify the currency ofinformation (i.e., whether information is current or outdated) and henceis able to provide more relevant and current information when asked.

Domain machine learning algorithm 1760 is configured to receivefacilities-related documents from document ingester 1750 which mayinclude manual scans of operator's manuals, technical specifications,data sheets, procedures, and similar materials or already-digitizedversions of such documents (e.g., in Portable Document Format (PDF)form) or materials obtained from the Internet from document ingester's1750 web crawler. Domain machine learning algorithm 1760 may alsoreceive information about the facility itself from a facilities database(e.g., facilities database 540) such as the types and numbers ofequipment installed, the operating capacities of such equipment,operating limits, alarms, and thresholds, processing capacities,nameplate ratings, etc.

Domain machine learning algorithm 1760 is connected to SCADA integrationmodule 1770 which allows ingestion of real-time, near-real-time, orstored operational data from equipment, sensors, logs, or otherdata-producing devices associated with a facility. Domain machinelearning algorithm 1760 is configured to incorporate such operationaldata and any learned information or insights into knowledge graph 1740.Outputs of domain machine learning algorithm 1760 may include automatedcontrol recommendations for equipment which are converted by SCADAintegration module 1770 to equipment control signals.

SCADA integration module 1770 has functionality similar to contextmachine learning algorithm(s) 330 of other embodiments. SCADAintegration module 1770 is configured to receive and transmit data andcontrol commands between a facility's SCADA system and robotic processautomater (RPA) 1700. As described above, the core of operationalcontrol of modern industrial facilities are supervisory control and dataacquisition (SCADA) systems. SCADA systems comprise programmable logiccontrollers (PLCs) and remote terminal units (RTUs) which aremicrocomputers that are connected with the equipment 140 and sensors 150throughout the facility. Depending on their configuration, the PLCs andRTUs acquire data from equipment, control equipment, or both. The PLCsand RTUs are connected to SCADA software that allows for monitoring of,and control of, the equipment, allowing for sophisticated,computer-controlled processes to be implemented on the equipment.

FIG. 18 is an exemplary knowledge graph of the type that may be used byan industrial virtual assistant with robotic process automation. In thisexample, a very simple water system is shown comprising a pump A 1810, avalve B 1820, a pipe C 1830, and a pipe D 1840, which are entities inthe knowledge graph. Pump A 1810 has several descriptor nodes associatedwith it connected by labeled edges which further describe the pump. Forexample, pump A 1810 is a high-pressure water pump 1811 which tends tooperate at 2 psi above its nominal (or nameplate) pressure 1812, andwhich is less than 10 years old 1813. Similarly, valve B 1820 isdescribed by a node and edge as being a two-way diverter valve. Pump A1810 is connected to valve B 1820, which is further connected to pipe C1830 and pipe D 1840. Pipe C is described by a node and edge as beingundersized for its intended purpose 1831 and pipe D is described by anode and edge as being right-sized for its intended purpose. A machinelearning algorithm run on this knowledge graph may make the inferencethat when water is diverted through valve B 1820 to pipe C 1830, thepressure at pump A 1810 may be higher than expected due to theundersizing 1831 of pipe C 1830 for its purpose. Additional contextadded to the graph may explain that the average of 2 psi above normal1812 of pump A 1810 is due to the period of time when valve B 1820 isswitched to pipe C 1830. Taken together, this knowledge graph provides acomprehensive set of knowledge of this simple system, and the conceptmay be scaled up to systems of any size.

FIG. 19 is a diagram showing an exemplary application of a roboticprocess automater aspect of an industrial virtual assistant. In thisexample, a robotic process automater (RPA) 1700 is applied to a watersystem. RPA 1700 uses both deductive learning 1910 and inductivelearning 1920 to evaluate the system, generate new information andinsights, respond to queries, and automate processes. Deductive learninginvolves taking general data and forming specific conclusions byformulating a hypothesis and testing the hypothesis against the data.Inductive learning involves taking specific data and forming generalconclusions by recognizing patterns within the data. In this example,RPA 1700 takes general data about the system such as “all asbestoscement (AC) pipe should be prioritized for replacement” 1911, “Zone Ahas the most asbestos cement (AC) pipe by length” 1912, and “Zone B hasthe highest percentage of AC pipe” 1913, stores that information inknowledge graph 1740, and runs machine learning algorithms on the datato obtain inferences and further insights to enable responses tospecific queries such as “what pipe material is likely to be replaced?”1914, “which zones should be prioritized for replacement?” 1915, and“what is the dominant pipe material in Zone B?” In this example, RPAfurther takes specific data about the system such as pipe burst eventsand analyzes them to form general conclusions about the system. Here,the specific data about pipe bursts includes Pipe Burst 1 data 1921having the characteristics of being in Zone A, having a pressure drop,being associated with acoustic noise, having water discolorationcomplaints from consumers, and being made of AC pipe, Pipe Burst 2 data1922 having the characteristics of being in Zone A, having a pressuredrop, being associated with acoustic noise, and being made of AC pipe,Pipe Burst 3 data 1923 having the characteristics of being in Zone B,having a pressure drop, and being associated with acoustic noise. RPA1700 stores that information in knowledge graph 1740, runs machinelearning algorithms on the data to obtain inferences and furtherinsights to create general conclusions about the system that can be usedto respond to queries such as “what is the dominant indicator of pipebursts in Zone A?” 1924, “what is the dominant indicator of pipe burstsin all zones?” 1925, and “is discoloration common to all pipe bursts?”1926. The inferences and insights gained from the deductive learning1910 and inductive learning 1920 are also stored in the knowledge graph,and allow RPA 1700 to provide context-specific and domain-specificmeaningful responses to queries or to automate controls.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of the aspectsdisclosed herein may be implemented on a programmable network-residentmachine (which should be understood to include intermittently connectednetwork-aware machines) selectively activated or reconfigured by acomputer program stored in memory. Such network devices may havemultiple network interfaces that may be configured or designed toutilize different types of network communication protocols. A generalarchitecture for some of these machines may be described herein in orderto illustrate one or more exemplary means by which a given unit offunctionality may be implemented. According to specific aspects, atleast some of the features or functionalities of the various aspectsdisclosed herein may be implemented on one or more general-purposecomputers associated with one or more networks, such as for example anend-user computer system, a client computer, a network server or otherserver system, a mobile computing device (e.g., tablet computing device,mobile phone, smartphone, laptop, or other appropriate computingdevice), a consumer electronic device, a music player, or any othersuitable electronic device, router, switch, or other suitable device, orany combination thereof. In at least some aspects, at least some of thefeatures or functionalities of the various aspects disclosed herein maybe implemented in one or more virtualized computing environments (e.g.,network computing clouds, virtual machines hosted on one or morephysical computing machines, or other appropriate virtual environments).

Referring now to FIG. 20 , there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one aspect, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one aspect, a computing device 10 may be configuredor designed to function as a server system utilizing CPU 12, localmemory 11 and/or remote memory 16, and interface(s) 15. In at least oneaspect, CPU 12 may be caused to perform one or more of the differenttypes of functions and/or operations under the control of softwaremodules or components, which for example, may include an operatingsystem and any appropriate applications software, drivers, and the like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some aspects, processors 13 may include speciallydesigned hardware such as application-specific integrated circuits(ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a particular aspect, alocal memory 11 (such as non-volatile random access memory (RAM) and/orread-only memory (ROM), including for example one or more levels ofcached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QUALCOMMSNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one aspect, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (Wi-Fi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 20 illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe aspects described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one aspect, a single processor 13 handles communications aswell as routing computations, while in other aspects a separatededicated communications processor may be provided. In various aspects,different types of features or functionalities may be implemented in asystem according to the aspect that includes a client device (such as atablet device or smartphone running client software) and server systems(such as a server system described in more detail below).

Regardless of network device configuration, the system of an aspect mayemploy one or more memories or memory modules (such as, for example,remote memory block 16 and local memory 11) configured to store data,program instructions for the general-purpose network operations, orother information relating to the functionality of the aspects describedherein (or any combinations of the above). Program instructions maycontrol execution of or comprise an operating system and/or one or moreapplications, for example. Memory 16 or memories 11, 16 may also beconfigured to store data structures, configuration data, encryptiondata, historical system operations information, or any other specific orgeneric non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device aspects may include non-transitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnon-transitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some aspects, systems may be implemented on a standalone computingsystem. Referring now to FIG. 21 , there is shown a block diagramdepicting a typical exemplary architecture of one or more aspects orcomponents thereof on a standalone computing system. Computing device 20includes processors 21 that may run software that carry out one or morefunctions or applications of aspects, such as for example a clientapplication 24. Processors 21 may carry out computing instructions undercontrol of an operating system 22 such as, for example, a version ofMICROSOFT WINDOWS™ operating system, APPLE macOS™ or iOS™ operatingsystems, some variety of the Linux operating system, ANDROID™ operatingsystem, or the like. In many cases, one or more shared services 23 maybe operable in system 20 and may be useful for providing common servicesto client applications 24. Services 23 may for example be WINDOWS™services, user-space common services in a Linux environment, or anyother type of common service architecture used with operating system 21.Input devices 28 may be of any type suitable for receiving user input,including for example a keyboard, touchscreen, microphone (for example,for voice input), mouse, touchpad, trackball, or any combinationthereof. Output devices 27 may be of any type suitable for providingoutput to one or more users, whether remote or local to system 20, andmay include for example one or more screens for visual output, speakers,printers, or any combination thereof. Memory 25 may be random-accessmemory having any structure and architecture known in the art, for useby processors 21, for example to run software. Storage devices 26 may beany magnetic, optical, mechanical, memristor, or electrical storagedevice for storage of data in digital form (such as those describedabove, referring to FIG. 20 ). Examples of storage devices 26 includeflash memory, magnetic hard drive, CD-ROM, and/or the like.

In some aspects, systems may be implemented on a distributed computingnetwork, such as one having any number of clients and/or servers.Referring now to FIG. 22 , there is shown a block diagram depicting anexemplary architecture 30 for implementing at least a portion of asystem according to one aspect on a distributed computing network.According to the aspect, any number of clients 33 may be provided. Eachclient 33 may run software for implementing client-side portions of asystem; clients may comprise a system 20 such as that illustrated inFIG. 21 . In addition, any number of servers 32 may be provided forhandling requests received from one or more clients 33. Clients 33 andservers 32 may communicate with one another via one or more electronicnetworks 31, which may be in various aspects any of the Internet, a widearea network, a mobile telephony network (such as CDMA or GSM cellularnetworks), a wireless network (such as Wi-Fi, WiMAX, LTE, and so forth),or a local area network (or indeed any network topology known in theart; the aspect does not prefer any one network topology over anyother). Networks 31 may be implemented using any known networkprotocols, including for example wired and/or wireless protocols.

In addition, in some aspects, servers 32 may call external services 37when needed to obtain additional information, or to refer to additionaldata concerning a particular call. Communications with external services37 may take place, for example, via one or more networks 31. In variousaspects, external services 37 may comprise web-enabled services orfunctionality related to or installed on the hardware device itself. Forexample, in one aspect where client applications 24 are implemented on asmartphone or other electronic device, client applications 24 may obtaininformation stored in a server system 32 in the cloud or on an externalservice 37 deployed on one or more of a particular enterprise's oruser's premises. In addition to local storage on servers 32, remotestorage 38 may be accessible through the network(s) 31.

In some aspects, clients 33 or servers 32 (or both) may make use of oneor more specialized services or appliances that may be deployed locallyor remotely across one or more networks 31. For example, one or moredatabases 34 in either local or remote storage 38 may be used orreferred to by one or more aspects. It should be understood by onehaving ordinary skill in the art that databases in storage 34 may bearranged in a wide variety of architectures and using a wide variety ofdata access and manipulation means. For example, in various aspects oneor more databases in storage 34 may comprise a relational databasesystem using a structured query language (SQL), while others maycomprise an alternative data storage technology such as those referredto in the art as “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLEBIGTABLE™, and so forth). In some aspects, variant databasearchitectures such as column-oriented databases, in-memory databases,clustered databases, distributed databases, or even flat file datarepositories may be used according to the aspect. It will be appreciatedby one having ordinary skill in the art that any combination of known orfuture database technologies may be used as appropriate, unless aspecific database technology or a specific arrangement of components isspecified for a particular aspect described herein. Moreover, it shouldbe appreciated that the term “database” as used herein may refer to aphysical database machine, a cluster of machines acting as a singledatabase system, or a logical database within an overall databasemanagement system. Unless a specific meaning is specified for a givenuse of the term “database,” it should be construed to mean any of thesesenses of the word, all of which are understood as a plain meaning ofthe term “database” by those having ordinary skill in the art.

Similarly, some aspects may make use of one or more security systems 36and configuration systems 35. Security and configuration management arecommon information technology (IT) and web functions, and some amount ofeach are generally associated with any IT or web systems. It should beunderstood by one having ordinary skill in the art that anyconfiguration or security subsystems known in the art now or in thefuture may be used in conjunction with aspects without limitation unlessa specific security 36 or configuration system 35 or approach isspecifically required by the description of any specific aspect.

FIG. 23 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to peripherals such as a keyboard49, pointing device 50, hard disk 52, real-time clock 51, a camera 57,and other peripheral devices. NIC 53 connects to network 54, which maybe the Internet or a local network, which local network may or may nothave connections to the Internet. The system may be connected to othercomputing devices through the network via a router 55, wireless localarea network 56, or any other network connection. Also shown as part ofsystem 40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various aspects, functionality for implementing systems or methods ofvarious aspects may be distributed among any number of client and/orserver components. For example, various software modules may beimplemented for performing various functions in connection with thesystem of any particular aspect, and such modules may be variouslyimplemented to run on server and/or client components.

The skilled person will be aware of a range of possible modifications ofthe various aspects described above. Accordingly, the present inventionis defined by the claims and their equivalents.

What is claimed is:
 1. A robotic process automation system, comprising:a computing device comprising a memory, a processor, and a non-volatiledata storage device; a knowledge graph stored on the non-volatile datastorage device, wherein the knowledge graph comprises nodes representingthings and edges representing relationships between the things; arobotic process automater comprising a first plurality of programminginstructions stored in the memory which, when operating on theprocessor, causes the computing device to: receive data about a facilityto be automated, the data comprising: one or more human inputs; one ormore documents; and one or more operational data streams from thesystem; store the received data in the knowledge graph; process aportion of the knowledge graph through a domain machine learningalgorithm to obtain an inference about the facility not contained in thereceived data; store the inference as part of the knowledge graph;receive a query about the facility related to the inference; process thequery and a portion of the knowledge graph through a context recognitionmachine learning algorithm to determine a context in which the query wasmade; process the context and a portion of the knowledge graph throughthe domain machine learning algorithm to: generate a response to thequery; and determine whether the inference can be generalized to thecontext of the query; and store the determination as part of theknowledge graph.
 2. The system of claim 1, wherein the query is anatural language query from a person and the response is provided as anatural language response using a natural language understanding (NLU)engine.
 3. The system of claim 2, wherein the natural languageunderstanding (NLU) engine comprises a natural language processingmachine learning algorithm trained in part on a domain-specific taxonomyfor a domain related to the facility.
 4. The system of claim 3, whereinthe natural language processing machine learning algorithm is furthertrained on information from the knowledge graph to enhance itsdomain-specific taxonomy recognition.
 5. The system of claim 1, whereinthe query is an automated query from a component of the facility and theresponse is a control recommendation sent to a supervisory control anddata acquisition (SCADA) module to generate a control signal.
 6. Thesystem of claim 1, wherein the information from the inferences about thefacility stored in the knowledge graph are analyzed over time toidentify long term patterns which have not been provided by the receiveddata about the facility.
 7. A method for operating a robotic processautomation system, comprising the steps of: storing a knowledge graph anon-volatile data storage device of a computing device comprising amemory, a processor, and a non-volatile data storage device, wherein theknowledge graph comprises nodes representing things and edgesrepresenting relationships between the things; using a robotic processautomater operating on the computing device to: receiving data about afacility to be automated, the data comprising: one or more human inputs;one or more documents; and one or more operational data streams from thesystem; store the received data in the knowledge graph; process aportion of the knowledge graph through a domain machine learningalgorithm to obtain an inference about the facility not contained in thereceived data; store the inference as part of the knowledge graph;receive a query about the facility related to the inference; process thequery and a portion of the knowledge graph through a context recognitionmachine learning algorithm to determine a context in which the query wasmade; process the context and a portion of the knowledge graph throughthe domain machine learning algorithm to: generate a response to thequery; and determine whether the inference can be generalized to thecontext of the query; and store the determination as part of theknowledge graph.
 8. The method of claim 7, wherein the query is anatural language query from a person and the response is provided as anatural language response using a natural language understanding (NLU)engine.
 9. The method of claim 8, wherein the natural languageunderstanding (NLU) engine comprises a natural language processingmachine learning algorithm trained in part on a domain-specific taxonomyfor a domain related to the facility.
 10. The method of claim 9, whereinthe natural language processing machine learning algorithm is furthertrained on information from the knowledge graph to enhance itsdomain-specific taxonomy recognition.
 11. The method of claim 7, whereinthe query is an automated query from a component of the facility and theresponse is a control recommendation sent to a supervisory control anddata acquisition (SCADA) module to generate a control signal.
 12. Themethod of claim 7, wherein the information from the inferences about thefacility stored in the knowledge graph are analyzed over time toidentify long term patterns which have not been provided by the receiveddata about the facility.